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Abstract 

In this paper we study the existence of locally most powerful invariant tests (LMPIT) for the problem of testing 
the covariance structure of a set of Gaussian random vectors. The LMPIT is the optimal test for the case of close 
hypotheses, among those satisfying the invariances of the problem, and in practical scenarios can provide better 
performance than the typically used generalized likelihood ratio test (GLRT). The derivation of the LMPIT usually 
requires one to find the maximal invariant statistic for the detection problem and then derive its distribution under 
both hypotheses, which in general is a rather involved procedure. As an alternative, Wijsman's theorem provides the 
ratio of the maximal invariant densities without even finding an explicit expression for the maximal invariant. We 
first consider the problem of testing whether a set of A^-dimensional Gaussian random vectors are uncorrelated or 
not, and show that the LMPIT is given by the Frobenius norm of the sample coherence matrix. Second, we study the 
case in which the vectors under the null hypothesis are uncorrelated and identically distributed, that is, the sphericity 
test for Gaussian vectors, for which we show that the LMPIT is given by the Frobenius norm of a normalized version 
of the sample covariance matrix. Finally, some numerical examples illustrate the performance of the proposed tests, 
which provide better results than their GLRT counterparts. 
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I. Introduction 

Testing the covariance structure of Gaussian vectors is one of the classic problems in multivariate statistical 
analysis (TJ, and is also commonly found in many applications, including sensor networks [2], cooperative networks 
with multiple relays (3)-||5), multiantenna radar detection |6) and, more recently, cognitive radio sensing with 
multiantenna spectrum monitors f7)-p0). Assuming Gaussian vector measurements, the problem in its most general 
formulation consists of testing whether the covariance matrix of the stack of all vectors is block-diagonal, that is, 
whether all pairwise cross-covariance matrices are zero or not. A case of particular interest results when all vector 
measurements share the same covariance matrix under the null hypothesis, in which case the test is a test of 
sphericity for Gaussian vectors. 

In practice, the likelihood depends on unknown parameters and the hypotheses are therefore composite. In the 
absence of a uniformly most powerful test (UMPT), most approaches consider the generalized likelihood ratio test 
(GLRT). However, despite its simplicity, it is known that the GLRT is not optimal in the Neyman-Pearson sense 
[[I]. Furthermore, although the GLRT tends asymptotically to be optimal fTT) , its performance may degrade for 
practical scenarios such as those typically found in cognitive radio applications, which are characterized by close 
hypotheses (low signal-to-noise ratio) and small sample sizes. When the problem exhibits symmetries or invariances, 
a reasonable approach consists in focusing on the class of tests satisfying the required invariances under a suitable 
group of transformations. Optimal invariant tests, which depend only on the maximal invariant statistic for the 
problem, are called uniformly most powerful invariant tests (UMPIT). Nevertheless, in many practical situations, 
such as those considered in this paper, the UMPIT does not exist (the maximal invariant statistic is vector-valued), 
and we have to resort to further refinements. Thus, by focusing on the challenging case of close hypotheses, and by 
applying a Taylor's series approximation of the ratio of maximal invariant densities, it might be possible to avoid 
the dependence on unknown parameters, yielding the so-called locally most powerful invariant test (LMPIT). The 
main goal of this paper is to study the existence of LMPITs for testing the covariance structure of Gaussian vectors, 
a problem for which UMP or UMPI tests do not exist in general. 

In contrast to the conventional way of deriving a LMPIT, which is finding the maximal invariant statistic and 



then deriving its distribution under both hypotheses, in this paper we apply Wijsman's theorem |12]-|14|. This 
powerful theorem (cf. Section |ll| allows us to obtain the ratio of densities even without an explicit formulation for 
the maximal invariant statistic. 



A. Related works and main contributions 

The main theoretical work pertaining to the present paper was done in the seventies by John [15], who derived the 
LMPIT in the scalar case for testing whether a set of real Gaussian random variables are uncorrelated and identically 
distributed (i.e., all with the same variance), the so-called sphericity test for Gaussian variables. Another related work 
is p6) , where the author derived a locally minimax test for testing independence among real Gaussian vectors. 
Apart from ]T5") and fl6) , the vast majority of published work for testing the correlation structure of Gaussian 
random variables or random vectors with unknown parameters uses the GLRT. Specifically, the first generalized 
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likelihood ratio test dates back to the work by Wilks in the 30s |17|, where the author derived the GLRT for 
testing the null hypothesis that the covariance matrix of a set of real Gaussian random variables is diagonal vs. the 
alternative that it is positive definite and otherwise arbitrary. That is a test for correlation among Gaussian variables. 
In particular, the GLRT derived by Wilks is given by the Hadamard ratio of the sample covariance matrix, i.e., the 
ratio between the determinant of the sample covariance matrix and the product of the elements of its main diagonal, 
which may also be rewritten as the determinant of the sample coherence matrix. More recently, this problem was 
revisited in the field of signal processing for radioastronomy by Leshem and Van der Veen (T8|, (19), who derived 
the GLRT for circular complex Gaussian random variables. Interestingly, the determinant of the sample coherence 
matrix was also proposed in |20|, [21 1 based on a geometric interpretation of the correlation coefficient, which they 
referred to as generalized coherence (GC). The GLRT for sphericity was found by Mauchly [22]. These results 
have been extended to vector- valued Gaussian data in [ |23| , where the authors derived the GLRT for testing whether 
a covariance matrix of complex Gaussian vectors is block-diagonal or not. Not surprisingly, the GLRT for this 
problem is a generalized (block-based) Hadamard ratio of the sample covariance matrix. 

In this paper, we focus on the case of close hypotheses and study the existence of LMPITs for the covariance 



structure of Gaussian data. We extend |15| to the general case of vector- valued observations that may or may not 
have the same covariance matrix under the null hypothesis. Instrumental in deriving these results is the application 
of Wijsman's theorem, which requires identifying the invariances of each problem and integrating the distribution 
of the transformed observations under a measure on the corresponding group of transformations. Although we focus 
on the case of complex Gaussian vectors, which is motivated by its applications in radar and spectrum sensing 
problems, the presented results can be easily proved for real vectors, which is left as an exercise for the interested 
reader. Specifically, the main contributions of this work are the following: 

• Correlation Test: For vectors with different covariances under the null hypothesis, we show that the LMPIT 
is given by the Frobenius norm of the sample coherence matrix. The block diagram of the LMPIT is depicted 
in the lower part of Figure [T] where we can see that the whitening is different for each vector. We should also 
point out that the Frobenius norm of the coherence matrix, or similar squared-sum test statistics, have been 
previously proposed as approximations of the GLRT that might be computationally simpler or have some other 



advantage [18], (19) , (24), |25|. Here, we prove for the first time that the Frobenius norm of the coherence 
matrix is in fact the LMPIT for this problem. 

• Sphericity Test: When the vector-valued observations under the null hypothesis are independent and identically 
distributed, we show that the LMPIT is given by the Frobenius norm of a normalized version of the sample 
covariance matrix. Figure [T] also shows the block diagram of the LMPIT for this problem. However, in this 
case, the whitening is common for all vectors, and it is given by the inverse square root matrix of the average 
of the individual sample covariance matrices. 

• Local Irrelevance of the rank-structure: Interestingly, the previous results also hold for the signal-plus-noise 
model commonly used in many signal processing problems, regardless of the rank of the signal covariance 
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Fig. 1. Block diagram of the LMPIT for the testing problems considered in this paper. Whiten denotes a whitening step, that is, the input 
vector is multiplied by the inverse of the square root matrix that enters into the block, R. is a block that computes the sample covariance matrix 
of the input-vectors, whereas It-kk computes the covariance matrix of the fcth vector, 1/LJ] averages the input matrices, and || ■ || 2 denotes 
the squared Frobenius norm of the input-matrix. As can be seen, the main difference among the detectors is the whitening step. For testing 
correlation among Gaussian vectors, the whitening step is different for each vector, whereas for testing sphericity the same whitening is applied 
to each vector. 



matrix. From a practical point of view, this means that for close hypotheses the spatial structure does not play 
any role for detection purposes. In other words, the use of subspace-based estimation techniques does not help 
for detection, at low signal-to-noise ratios (SNR) and/or low sample support. 
The paper is organized as follows. Section |Il] presents a brief review of Wijsman's theorem and the use of 
invariance in detection problems. The LMPIT for testing whether L non-identically distributed TV-dimensional 
vectors are uncorrelated is introduced in Section [Til] whereas Section [TV] presents the LMPIT when the vector 
observations under the null hypothesis are identically distributed. The performance of the proposed detectors is 
illustrated by means of numerical simulations in Section [V] and Section VI summarizes the main conclusions of 
this work. 
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B. Notation 

In this paper we use bold-faced upper case letters to denote matrices, with elements Xk,i, bold-faced lower case 
letters for column vectors, and light-face lower case letters for scalar quantities. The superscripts ( ) T and (•)•" 
denote transpose and Hermitian transpose, respectively. The determinant, trace and Frobenius norm of a matrix A 
will be denoted, respectively, as dct (A), tr (A) and ||A||. The notation A £ C MxN (A e R MxN ) will be used 
to denote a complex (real) matrix of dimension M x N. For vectors, the notation x e C M (x e M A/ ) denotes 
a complex (real) vector of dimension M. The absolute value of the complex number x is denoted as \x\ and 
x ~ CAf(f4, R) indicates that x is a complex circular Gaussian random vector of mean and covariance matrix 
R. The expectation operator will be denoted as E[-] and ® is the Kronecker product. Il is the identity matrix of 
size L x L and l denotes the zero vector or the zero matrix (depending on the context) of sizes Lxl and Lx L, 
respectively. We use A 1 / 2 (A -1 / 2 ) to denote the Hermitian square root matrix of the Hermitian matrix A (A -1 ). 
Finally, oc stands for equality up to additive and multiplicative positive constant (not depending on data) terms and 
diagjy(A) is a block-diagonal matrix formed by the N x N matrix blocks on the diagonal of A, whereas diag(A) 
denotes the vector composed by the elements of the main diagonal of A. 

II. Invariant Tests: Wijsman's Theorem 
It is well known that the Neyman-Pearson detector, given by the likelihood ratio, is the optimal test for simple 



hypotheses (26 1, that is, for hypotheses whose likelihood is known. Nevertheless, when the likelihood depends on 
unknown parameters, the hypotheses are composite and the problem is far from trivial. In fact, an optimal detector 
for composite hypothesis, known as a uniformly most powerful test (UMPT) |26) , (27), only exists in a few fortunate 
simple cases. 

When a UMPT does not exist, a typical approach consists in focusing on the class of detectors invariant to some 
transformations suitable for the problem. Let us first introduce the concepts of invariance and maximal invariant 
statistic that will be used in the rest of the section. A detector is said to be invariant to the group of transformations 
g if (26), (28) 

T( 3 (x))=T(x), V 5 G)e<?, (1) 



where T(-) is the test statistic and x are the measurements. A statistic M(x) is maximal invariant if (26) , (28) 

M( 5 (x)) = M(x), (2) 

for all g(-) £ Q and 

M( Xl ) = M(x 2 ), (3) 

implies x 2 = g(x.i) for some g(-) G Q. Therefore, the maximal invariant statistic organizes the measurements into 
sets that provide a constant value of M(x), and these sets are known as orbits. Taking this into account, every 
invariant test may be written as a function of the maximal invariant statistic, 

T(x) = T(M(x)), (4) 
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and we can therefore restrict our attention to rules that are functions of M(x). Using this important concept, the 
conventional approach to obtaining a uniformly most powerful invariant test (UMPIT) is as follows p6[ : 

1) Identify the problem invariances and identify the corresponding transformation group. 

2) Find the maximal invariant statistic. 

3) Derive the densities of the maximal invariant statistic under each hypothesis. 

4) Obtain the detector based on the ratio of the densities of the maximal invariant statistic. 

5) Establish that the detector statistic has a monotone likelihood ratio 

This approach can be, for many problems, very difficult due to the need to obtain the distribution of the maximal 
invariant statistic under each hypothesis, which might itself be a complicated function of the data. Wijsman's 
theorem fT2)-fl4") provides an alternative (simpler) way to derive the ratio of densities of the maximal invariant 
statistic. The theorem states that, under some mild assumptions (presented later), the ratio of the maximal invariant 
densities is given by 

^ = f g p(g(x);H 1 )\det(J g )\dg 
J e P(ff( x );^o) |det {3 g )\dg' 
where p(yL;Hi) is the probability density function (pdf) of x under the hypothesis Hi, Q is the group of invariant 
transformations, 3 g denotes the Jacobian of the transformation g(-) E Q and dg is an invariant group measure, 
which in our problems may be taken as the usual Lebesgue measure. Despite its usefulness, Wijsman's theorem 
has received little attention in the information theory and signal processing communities and just a few works have 
exploited this result to derive invariant tests for particular problems p9)-p5). 

Stein (36) was the first to propose the idea of integrating over the group of transformations that describe the 
problem invariances. Later, the conditions for the validity of the theorem were studied in (T2J, (13) , [30) , p7)-p9j 
by Wijsman and other authors, who formally proved the theorem. For us, it suffices to consider the simplest set of 
conditions in (12) and (30). In particular, Q has to be a Lie group, a finite group or a composition of both and the 
observations have to belong to a linear Cartan Q -space, that is, a nonempty open subset (denoted as S) of Euclidean 
space such that, for every x E S, there exists a neighborhood V for which the closure of {#(•) £ Q : g(V) D V ^ 0} 
is compact. 

Unfortunately, for the testing problems considered in this paper, the ratio of the maximal invariant densities 
(obtained through Wijsman's theorem) still depends on an unknown parameter vector, which means that the UMPIT 
does not exist in general. For this reason, we will focus our attention on the case of close hypotheses and will 
study the existence of locally most powerful invariant tests (LMPIT). The main idea behind the LMPIT consists 
in applying a Taylor's series approximation of the likelihood ratio. When the lowest order term depending on the 
data is a monotone function of a scalar statistic, the detector is locally optimal. We will detail this procedure in the 
following sections for the tests considered in the paper. 

III. The LMPIT for Correlation of Gaussian Vectors 

Consider a set of L zero-mean circular complex jointly Gaussian vectors, xi,...,x,j,, of dimension N, i.e., 
x s ; e C N , i = 1, . . . , L. In this section, we address the problem of testing whether these vectors are correlated or not, 
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without assuming any particular structure for the covariance matrices. That is, we want to check if E [xfcXp] = 0^, 
Vfe 7^ We shall proceed by constructing the NL vector x = [x{ , . . . , x^] T , and defining its covariance matrix as 



R = B[xx ff ] 



Rn 

R 12 



Rl2 

R22 



RlL 

R2L 



R 



LL 



(6) 



with R e qlnxln an( j g ^jvxa/ Hence, the detection problem may be cast as the following test for the 
covariance structure of x, 

«i:ReS, 

(7) 

Ho ■ R G Db + , 

where § is the set of positive definite matrice^j and Db + is the set of block-diagonal matrices with positive definite 
blocks. As previously pointed out, we do not impose any particular covariance structure on the individual vectors, 
beyond uncorrelation among vectors under Ho- Then, the test becomes 

Hi :x~CM(p LN ,R.i), 
H :x~CAA(0 LA r,D), 

where Ri e § and D 6 Db + are two unknown covariance matrices. This test is rather general and encompasses, for 
instance, the problem of testing whether a set of L univariate non-stationary Gaussian time series are uncorrected 
(23). 



(8) 



A. Derivation of the LMPIT 

Given a set of M vector measurements, x[l], . . . , x[M], in this subsection we apply Wijsman's theorem to 
derive the LMPIT for the detection problem ([8]). To this end, it is first necessary to find the problem invariances. 
Specifically, this hypothesis test is invariant under the group of transformations 

Q= {.g:x^.g(x) = PGx,P = P ® I^P G P, G G D B } (9) 

where P is the set of permutation matrices of dimension L and Ob is the set of block diagonal invertible matrices. 
That is, the test is invariant under permutations of the vectors, and to a (possibly different) linear transformation 
of each of them. 

Let us also obtain an explicit expression for the maximal invariant under the transformations in d9j. This will 
help to identify the difficulties of the conventional approach to deriving invariant tests for this problem, and to 
further appreciate the value of Wijsman's theorem. It is well known that the sufficient statistic for this problem is 
the sample covariance matrix 

1 M 

R = — x[m]x ff [m]. (10) 

rn — l 

'The results may be extended to consider positive semi-definite matrices. 
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Now, taking into account the invariance of the problem to a multiplication by an invertible block-diagonal matrix, 
we may introduce the transformation y = D -1 / 2 x, whose covariance matrix is 



c = rr 1/2 RD- 1/2 



'12 



IN 



CiL 
C2L 



K -'2L 



IN 



(ID 



where D = diag A ,(R) and C is the coherence matrix. After this initial prewhitening stage, we can arbitrarily fix 
the order of the vectors, for instance, according to the determinants of the matrices C/c,;|^] 
Once the order has been fixed, consider the singular value decomposition (SVD) of Ci )2 

612 = U 12 diag ($ 12 ) Vg, 



(12) 

where £ 12 contains the canonical correlations between the first and second vectors. Thus, we may apply a unitary 
transformation of y belonging to Q, that diagonalizes C12 as follows: z = diag (Uf^, V12, 1, ■ ■ ■ , l) y. Additionally, 
the LQ decomposition of the first row and column partitions allows us to express R zz as 



R, 



l N diag(£ 12 ) L13Q13 

dia g(€l 2 ) l N V12C23 

*^23 V 12 L N 



(~)H t H 



^2L V 12 



LilQil 

V12C2L 
C3L 



(13) 



where ~Lki is a lower triangular matrix with non-negative real diagonal elements and Qki is a unitary matrix. 
Finally, we can apply another linear transformation to triangularize the first row and column partitions as follows 
w = diag (I, I, Q13, . . . , Qil) z - As an example, the final structure of R ww is shown in Figure [2] The following 
lemma summarizes the obtained result. 

Lemma 1: The maximal invariant statistic for the test ([8]) under the group of transformations |9]) is given by 



{^2} U {L£, I = 3, . . . ,L} U {Qi fc C fei Q« k = 2, ...,L,l = 3, ... ,L,l > fc} , 



(14) 



where £ 12 are the canonical correlations between the first and second vectors, and LkiQki is the LQ decomposition 
of (fc, Z)th block of R zz . 

Although we have been able to obtain an explicit expression for the maximal invariant statistic, it can be seen that 
it is a complicated function of the observations. Therefore, deriving its distribution under each hypothesis seems 
intractable. Additionally, Lemma 1 also allows us to conclude that, since for N > 1 or L > 2 the maximal invariant 
is a vector- valued function of the data, the UMPIT for this problem does not exist in general |40|. Finally, notice 
that the maximal invariant statistic does not only depend on the eigenvalues of the coherence matrix (or canonical 
correlations), as shown in the following example. 



2 The determinants will be different with probability one. 
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Example: Consider the following coherence matrices (L = 3, N = 1) 





1 


0.5 







1 





0.4 


Ci = 


0.5 


1 





, c 2 = 





1 


0.3 










1 




0.4 


0.3 


1 



which are not related by one of the transformations in the invariance group Q, i.e., they belong to different orbits. 
However, it can be easily verified that they share the eigenvalues (canonical correlations), given by 0.5, 1, 1.5. 
Due to the difficulties posed by the maximal invariant of this problem, we resort to Wijsman's theorem to derive 



the LMPIT. It is easy to check that Ob is a Lie group and, for M > N, the observation space S = 



•>LNxM 



is a 



linear Cartan -space. On the other hand, the group of permutations is finite and it is not therefore a Lie group. 
Then using the results in J30[ for finite groups, and taking into account that also the conditions in p2| are fulfilled, 
Wijsman's theorem may be applied. Specifically, the ratio of the densities of the maximal invariant is given by 
= Jo b det (R!)- M |det (G)|™ e -^ T ^^ a ) dG 
' E P / DB det(Dr M |dct(G)| 2M c- Mtr ( pTD - lpGftGH )rfG ' 
In order to derive the LMPIT, we must factor ( fT3] > as a function depending only on the observations and a function 
depending on the unknown parameters (Ri and D). Before proceeding, let us introduce the following lemma. 
Lemma 2: The ratio of the maximal invariant densities ( fT5j ) may be simplified as follows 

^oc]T / |det(G)| 2M e- MtI '( GGff )e- Q dG, 



(16) 



where 



(17) 



k,l=l 
k^l 
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Here, the matrix G k is the feth block of G and S fc; is the (k,l)th block of S = P T D S 1/2 SD S 1/2 P, where 
S = R^ 1 and D s = diag Ar (S). 

Proof: Considering the change of variable G — > GD -1 ' 2 , which belongs to the group of transformations Q, 
(jT3J becomes 

In words, this transformation shows that the detector does not depend on the individual sample covariances, but 
on the sample coherences. Taking into account that the blocks of the main diagonal of C are equal to the identity 
matrix, it is straightforward to prove that the denominator of ( fT8j ) does not depend on the observations. Finally, 
considering a new change of variable from the group of transformations G — > P T D S ^ 2 PG, the result follows. ■ 

It can be easily shown that, in general, ( [To] ) is a function of the unknown parameters, given by S, which allows us 
to conclude again that there does not exist a uniformly most powerful invariant test (UMPIT). The only exception 
is presented in the following theorem. 

Theorem 1: For L — 2 and N — 1, that is, two scalar observations, the UMPIT^ accepts Ho for small values of 

T(x[l],...,x[M]) = |c 12 |, (19) 

where cyi is the sample correlation coefficient. 

Proof: For scalar observations, ( [To} is an increasing monotone function of the maximal invariant statistic |ci2 1 - 

■ 

Due to the nonexistence of the UMPIT for vector-valued data, we shall focus on the challenging scenario of 
close hypotheses, and derive the locally most powerful invariant test (LMPIT), which is presented in the following 
theorem. 

Theorem 2: The LMPIT accepts "Ho for small values of 

T(x[l],...,x[A/]) = ||C|| 2 . (20) 

Proof: See Appendix |A| ■ 
For the particular case of scalar observations, that is N = 1, the Frobenius norm of the coherence matrix has been 



previously proposed as an ad-hoc approximation of the GLRT detector [18], |19|. Additionally, in |23) the authors 
noticed experimentally that the Frobenius norm presents better performance than the determinant of the coherence 
matrix (which is the GLRT for this problem), mainly for close hypotheses and/or low sample sizes. Following a 
rigorous approach, in this section we have shown that the Frobenius norm of the coherence matrix is not just a 
reasonable approximation of the GLRT, but the LMPIT; that is, the optimal invariant detector for close hypotheses. 



Actually, it is shown in |27 1, using a different approach, that the sample correlation coefficient is the uniformly most powerful unbiased test. 
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B. Latent Signal Model 

Many signal processing, communications and econometrics problems use the well known signal-plus-noise or 
latent signal model. In these cases, the hypothesis test has further structure that, in principle, could be exploited to 
improve the detector performance. In particular, the test in this case is given by 

Hi : x ~ CAT{Oln, HH ff + D), ^ 
H :x~CAA(0 LJV ,D), 

where H e £LNxp j s ^ un ] £nown channel matrix and D £ Db + is the unknown block-diagonal noise covariance 
matrix. The main difference with our original problem is that in ( |2T| the covariance matrix under Hi has more 
structure than in (j8j. Specifically, it is the sum of a (possibly) rank-deficient matrix and the noise covariance matrix. 
As it turns out, the derivation of the GLRT for this problem is pretty involved, even for the simple case of TV = 1 
j9). In particular, the authors of (9j, considering scalar observations, showed that there is no closed-form GLRT. On 
the other hand, it is easily proved that the additional structure imposed in Ri does not modify the invariances of the 
testing problem in (JS) and the maximal invariant remains the same, which has the following important consequence. 

Remark 1: The LMPIT for the test given in ( fH) is also the Frobenius norm of the coherence matrix given by 
( |20p , which means that, for close hypotheses, the additional spatial structure in the signal subspace is irrelevant for 
optimal detection. 

The optimality of ignoring the spatial structure can be explained as a direct consequence of the fact that, for 
close hypotheses (e.g., low SNRs or small sample size), errors in the estimation of the signal subspace are likely 
to occur, thus degrading the performance of the detector. Consequently, it makes more sense in this situation to 
average out the unknown parameters rather than using their maximum likelihood estimates. 

IV. The LMPIT for Sphericity of Gaussian Vectors 

In this section, we consider a more restrictive null hypothesis under which the vectors are also identically 
distributed. Specifically, we consider the following test 

Hi :x~C7V(0 LJV! Ri), 

Ho :x^CAA(O iAr ,I L ®R ), 
where Rx e § is the LN x LN unknown covariance matrix under Hi and R £ § is the N x N unknown 
covariance matrix under Hq. Hence, this hypothesis test generalizes the well-known test for sphericity of Gaussian 
variables (22). For this problem the group of invariant transformations is 

G = {g : x -> g(x) = (Q®G)x,QeQ,GeG}, (23) 

where Q is the set of L x L unitary matrices and G is the set of N x N invertible matrices. That is, the problem 
is invariant under the same linear transformation of all vectors and any unitary combination of them. The maximal 
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invariant for this problem is again a complicated vector- valued function of the observations^] which does not allow 
us to derive the distributions under each hypothesis and also shows that the UMPIT does not exist in general. In 
the following we use Wijsman's theorem to obtain the LMPIT for this problem. 

The group of transformations Q is a Lie group, and for M > N the observation space S = C LNxAI is a linear 
Cartan 5-space. Hence, the conditions of Wijsman's theorem are fulfilled, which allows us to express the ratio of 
the maximal invariant densities as follows 

" J q J G |det (G) |^M e _Mtr((l®R - 1 )GRG-) dGdQ ' 

where we have introduced the matrix G = Q G to simplify notation and dropped the size of the identity matrix. 
First, let us present a simpler expression for Jz? , given in the following lemma. 
Lemma 3: The distribution ratio is given by 

JSfoc / / |dct(G)| 2LM c- Q dGdQ, (25) 
Jq Jg 

where 

a = Aftr (SGRG H ) . (26) 

The normalized sample covariance matrix is 

R = (I®R )- 1/2 R(I«>Ror 1/2 , (27) 

and the matrix S is given by 

S = (I® S)- 1/2 S(I<g) S)~ 1/2 , S=R^ 1 . (28) 

Ro = 1/L X)fc=i R-kk is the ML estimate of Ro and S = 1/L X)t=i ^kk is the mean of the diagonal blocks of S. 
Proof: After applying the change of variable given by G — > GR 1 ^ 2 , the ratio |24) becomes 



a / Q / G |det(G)| e V dGdQ ^ 

f Q f {J |det(G)| 2Z ' M e- Mtr (N(^Ro 1 G)]i) dGdQ - 

1/2 

To get rid of the denominator, let us apply in the integral of the denominator the change of variable G — > R G, 



which yields 



ifoc / / |det(G)| 2LM e- Mtr ( s6li6ff )-iGdQ. (30) 



Finally, applying G — >• S G concludes the proof. 



Analogously to the previous case, < |25j > is in general a function of the unknown parameters and, therefore, the 
UMPIT does not exist. In this case, the exception is presented in the following theorem, which was previously 
proved in [40 1 following the conventional approach. 

4 We leave the derivation of the maximal invariant statistic as an exercise for the interested reader. 
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Theorem 3: For L = 2 and N = 1, that is two scalar observations, the UMPIT accepts Ho for small values of 

T(x[l] > ...,x[Af])=| ll (31) 

where £i is the largest eigenvalue of R, or equivalently, the largest eigenvalue of R normalized by tr(R). 

Proof: For L — 2 and N = 1, G becomes a scalar, namely g. Therefore, taking into account the eigenvalue 
decomposition (EVD) of S and R, the ratio (|2~5|) may be rewritten as 



J^oc / / | 5 | 4M e- M lsl 2 n, t A t C s | 9t , s | 2 dffdQ; (32) 



where A t is the tth eigenvalue of S and q t s denotes the (t, s)th element of Q. Finally, noting that tr(R) = £1+^2 = 
2, it is easy to show that ( f32j > is an increasing monotone function of £1. ■ 

Excluding the particular case presented in the previous theorem, we have to focus on the case of close hypotheses 
and derive the LMPIT, which is presented next. 

Theorem 4: The LMPIT accepts Ho for small values of 

T(x[l],...,x[M]) = ||R|| 2 . (33) 

Proof: See Appendix [B] ■ 
To conclude this subsection, let us consider again the latent signal model, which results in the following hypothesis 
test 

Hi :x~CAf(0 LN ,HH H +I L ®R ), 
Ho :x~a/V(0 LJV ,lL®Ro). 



The group of invariant transformations for the above problem is still given by ( |23| >. Hence, we can conclude that 
the LMPIT for the detection problem p4| is also given by ( [33"} . This interesting result shows again that, under 
close hypotheses, the additional rank structure of the covariance matrix is irrelevant for optimal detection. 

V. Numerical Results 

In this section we illustrate the performance of the LMPITs in several scenarios. In all examples the covariance 
matrices are as follows 

W 1 :R=(F £ n z Ff)<8l^ ) 
Ho ■ R = Iln, 

where F^ is the Fourier matrix of dimensions L x L and fi^ is an L x L diagonal matrix containing the 
canonical correlations, i.e. the singular values of the coherence matrix. In particular, we have considered L canonical 
correlations equispaced between 0.5 and 1.5. It is important to point out that, due to the problem invariances, there 
is no loss of generality by considering the identity covariance matrix under the null hypothesis. 

In the first scenario we consider the performance of the LMPIT given by the Frobenius norm of the coherence 
matrix and compare it to that of the GLRT, which accepts Ho for large values of 

T(x[l],...,x[M]) =logdet(c) . (36) 
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Specifically, this example considers L = 10 vector measurements of dimension N = 4, and generates the receiver 
operating characteristic (ROC) curves for M — 40, 55, 70, 85, and 100 samples. The ROC curves are shown in 
Figure [3] where we can see that the performance of the LMPIT is consistently better than that of the GLRT for 
this example with close hypotheses. Moreover, the performance gap increases for small values of M, as Figure [4] 
clearly shows. 

In the second scenario we evaluate the performance of the LMPIT given by ( [33) , and that of the GLRT for the 
sphericity test for Gaussian vectors. In particular, the GLRT accepts Ho for large values of 

T(x[l],... J x[M])=logdet(i). (37) 

For this example, Figures [5] and [6] show the ROC curves and the probability of missed detection, p m , vs. M for 
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Fig. 5. ROC curves for the GLRT and the LMPIT for testing sphericity among Gaussian vectors. The parameters of the experiment are: 
L = 10, N = 4 and M = 40, 55, 70, 85, and 100. 




Fig. 6. Probability of missed detection vs. M for the GLRT and the LMPIT for testing sphericity among Gaussian vectors. The parameters 
of the experiment are: L = 10, N = 4 and we fix pf a = 10 . 



an experiment with L = 10 sets of vectors, N = 4 and M — 40, 55, 70, 85, and 100 observations, where we draw 
conclusions similar to those in the previous example. 

Finally, let us make some comments on threshold selection. Similarly to pT) , and exploiting the invariances of the 
problem, the thresholds may be obtained using numerical simulations for the particular case Ro = I. These values 
will still be valid for any other choice of Ro. As an example, Figure [7] shows the empirical cumulative distribution 
function of the LMPIT and the GLRT for the first scenario, where we can see that the distribution of the (normalized) 
LMPIT statistic is almost independent of M. Moreover, taking into account that logdet(C) oc ||C|| 2 for C ss I 
(TS), [23 1, we may use the results for the asymptotic distribution of the GLRT provided by Wilks' theorem fTT) . 
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Fig. 7. Empirical CDF for the GLRT and the LMPIT for testing correlation among Gaussian vectors. The parameters of the experiment are: 
L = 10, N = 4 and M = 40, 55, 70, 85, and 100. Red solid line: LMPIT. Blue dashed line: GLRT (Curves far away from the Chi-squared 
approximation correspond to smaller values of M). Black dashed-dotted line: Wilks' approximation. 



Then, as M — > oo, the distribution of the LMPIT converges to a Chi-squared distribution 

(M||C|| 2 -LNM)^°x 2 (L 2_ L)N 2, (38) 
(M\\R\\ 2 -LNM)^ xf L 2^ 1)N ,. (39) 
In Figure [7] we can see how accurate the approximation is for this example. 

VI. Conclusions 

In this paper, we have derived two locally optimal invariant tests for the covariance structure of Gaussian random 
vectors. We have focused on the case of close hypotheses, which is of interest in many practical problems such as 
cognitive radio sensing, where detection must be performed in low signal-to-noise ratio (SNR) scenarios. Locally 
most powerful invariant tests (LMPIT) for this problem were only known for the case of i.i.d. scalar observations 
(with the same variance under "Ho), since in this situation the maximal invariant statistic is a simple function of the 
observations and its distributions can be analytically characterized. For vector-valued data, the maximal invariant 
statistic is typically a complicated function of the observations. To avoid these difficulties, in this paper we have 
applied Wijsman's theorem to obtain the ratio of the maximal invariant densities by integrating over the group of 
transformations defining the invariances of the problem. Based on this theorem, we have proved that the LMPIT for 
testing whether a set of Gaussian vector measurements are correlated is given by the Frobenius norm of the sample 
coherence matrix. For random vectors identically distributed under the null hypothesis, that is, the sphericity test for 
Gaussian vectors, we have also proved that the LMPIT exists, and is given by the Frobenius norm of a normalized 
sample covariance matrix. The invariances of the problems and our results do not depend on any additional rank 
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structure the covariance matrices might have. In this way, an interesting conclusion from a practical point of view 
is that the additional spatial structure is locally irrelevant for optimal detection. 

Appendix A 
Proof of Theorem|2] 

For close hypotheses (or low correlation), the inverse covariance matrix satisfies S s» I =>■ a ps 0. Therefore, 
applying a second order Taylor's series approximation of e~ a around a = we get 



& oc V f /3(G) (a 2 - 2a) dG, 



(40) 



where /3(G) = |det (G)| 2M e ~ Mtr ( GGif ) is a positive term only depending on G. Let us now consider the linear 
term, which is given by 

ificc^^ /3(G)tr(s fei G ; Q fe Gf)rfG. (41) 



Applying the change of variable Gfe — > — G&, for all values of k, it is easy to check that all the integrals are equal 
to their opposites, so they must be zero. In consequence, the linear term must be zero. Using similar arguments, 
we can get rid of the cross-product terms in a 2 and rewrite «5f as follows 

L 

K E E / tr ' (SfeiGzQfcGf ) dG. (42) 

Performing the singular value decomposition (SVD) of Sk,i and C^, the trace may be rewritten (without modifying 
the value of the integral) as 

N 

tr (s H G,C,*Gf ) - ]T XfifgL (gt)* , (43) 



s,t=l 

[kl 



where \f is the ith singular value of Ski ar, d 9t s denotes the (£, s)th element of G;. Therefore, ( |42) becomes 

2 



^«EE/ ^( g ) ( E Wsi,. (ft)*) ^ G - (44) 

Expanding the square, we get terms proportional to 

g\J^ s ,(gl s gl s ,)* , (45) 

and, using again a change of variable g\ s — > —g\ s for t ^ t' or s ^ s', we can discard the cross terms, yielding 

l n 2 
i^«EE E (W) A, (46) 

P fc,I=la,t=l 

where 

' ' r/CJ. (47) 



A = / /3(G)Re (5^(5^) 
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It is easy to see that A takes the same value for every required combination of the values of k, I, s, t and, therefore, 
the ratio S£ becomes 

; V v if 



L N 



^EEE(^) E(e) = E 

P k,l=l t=l s=l fc,/=l 

k>l k>l 





2 












Gfc 



(48) 



The sum over the set of permutations of the blocks of S cannot depend on the actual indexes, k or I. In particular, 
recalling that the blocks of the main diagonal of S are equal to the identity, we have J2p \\^ki\\ 2 oc ||S|| 2 , which 
allows us to write (RSI as 

L 

ifoc Y, ll^fcll 2 - (49) 

k,l—l 
k>l 

Finally, we conclude the proof noting that ||C/A;|| 2 — ||Cfc;|| 2 and ||C«|| 2 = N. 

Appendix B 
Proof of TheoremHJ 

Focusing on the case of close hypotheses (S ~ I) we can write 

JSfoc / / /3{G)(a 2 -2a)dGdQ, (50) 



where 

a = MtT ^G H SGR.) , (51) 
and S = S — Iln- Now, we shall consider the linear term, which may be written as 

/3(G) tr ( G H SGn) dGdQ = tr (#R) , (52) 



where 

*= / / (3(G)G H SGdGdQ. (53) 



Lemma 4: The matrix ^ is proportional to the identity. 

Proof: Consider the change of variable Q — » QD, where D = diag(d) is one out of the 2 L different diagonal 
matrices with ±1 entries. Then, we can write 

*= / / /3(G) (B<S)I N )G H SG(T) ®I N ) dGdQ = (D ® I N ) V (D ® I N ) . (54) 



Since the previous expression is valid for all the matrices D, we can conclude that \& must be block-diagonal. Let 
us now consider the change of variable Q — > QP, where P € P is one out of the LI different permutation matrices, 
yielding 

*= f I /3(G) (P <g) l N ) G H SG (P ® Iat) t dGdQ = (P ® I N ) * (P ® I N f , (55) 
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and allows us to conclude that the blocks in the diagonal of \& are identical. In order to obtain a matrix proportional 
to the identity, we follow the previous lines exploiting the transformations G. Therefore, introducing the change of 
variable G — » GDP, with D a diagonal matrix with ±1 entries and P a permutation matrix, we have 



*=/ / /3(G)(I L <g>DP)G"SG(I L ®DP) J dGdQ = (I L ® DP) * (I L ® DP) , (56) 

Jq Jg 

and we conclude that & oc Iln- ■ 
Now, taking into account the previous lemma and that tr(R) = LN, it is easy to see that 

tr (*R) oc tr (r) = LN, (57) 

and, therefore, the linear term does not depend on data. Hence, the density ratio becomes 

jSfcx / / /3(G) tr 2 (g^SGR) dGdQ, (58) 

which is simplified in the following lemma. 

Lemma 5: The density ratio of the maximal invariant statistic can be rewritten as 

J5f oc / / /3(G) tr 2 (G H SGR) dGdQ 

JqJg ■ ' 

n n L L L n n 

= / //3(G) ^tr(M fcfc R fefe ) dGdQ + 4^ /3(G)Re 2 {tr (M ife R fei ) } dGdQ, (59) 

JqJG \_k=l J k=l l=k+l "' G 

where M = G H SG. 

Proof: We shall decompose the trace as follows 

L L L 

tr (G H SGR) = tr (M fefc R fcfe ) + 2 Re { tr ( M ^ki) } , (60) 

and we may therefore write 



k=l k=ll=k+l 



[ [ /3(G) tr 2 (MR) dGdQ, (61) 

JqJg ^ ' 



as a Unear combination of terms of the form 



J J /3(G)Re {tr (M m „R„ m ) } Re {tr (M lk & k l) } dGdQ. (62) 

The above integrals can be simplified by introducing the change of variable Q — > QD (again with D = diag(d), 
and d = [di, . . . , d^] T a vector with ±1 elements), which yields 



tp = d, 



l d n d k di J J /3(G)Rc {tr (M m „R nm ) } Re {tr (M /fc R fe; ) } dGdQ. (63) 



Finally, the proof concludes by taking into account that the above equation must be valid for any choice of d. ■ 
Now, we shall split the proof in two parts, one for each term on the right hand side (RHS) of {59) , The first is 

given in the following lemma. 

Lemma 6: The first integral in the RHS of d59|» is given by 



L 2 



k=l 



(64) 
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with c > a constant term. 

Proof: Considering the change of variable Q — > QP^, where P T is the permutation matrix defined by the 
re-ordering ir, we may write 

v l i 2 r l i 2 

T= f f 13(G) Vtr(M fcfc A fcfc ) dGdQ = f f /3(G) V tr (M fcfe R, [fc]i7r[fe] ) dGdQ, (65) 

JQJG |_fc=l V 7 J JQJG [ fe=1 V 7 _ 

where R^j^fe] denotes the /cth block in the diagonal after permutation w. Let us now expand the square 

r= f //3(G)Vtr 2 (M fefc R 7r[fc] . T[fe] )rfGrfQ 
JqJg k=l v 7 

+ / / /3(G) ^ ^ tr (m k kK[kU[k\) tr (M H R x[;]i7r[;] ) dGdQ. (66) 

jqjg fe=i ;=i 

Analogously to the previous cases, the value of the integral T does not depend on the particular choice of the 
permutation. Thus, we can average over the LI possible permutations, and after some simplifications we get 

1 f r L L ^ 

r = - / / /3(G) E tr ' { M kkR mm ) dGdQ 

J®_Jg fc=lm=1 

n n L L L ^ ^ 

+ jY^y / / (3(G) E E E tr (MfcfcRmm) tr (M„ (lI n - R mm ) ) dGdQ, (67) 
where we have used J2k=i ^ kk = ^JV- Expanding the second term yields 

L L 



1 /* 

z 



/ £( G ) E E tr2 ( M ^™«) dGd Q 

fc=lm=l 

L L L ^ 

+ j— / / /3(G) E E E tr (M fefc R mm ) tr (Ma) dGdQ 
Jo Jg l=1 m=1 

L L L 

^EEE* 1 ( Mfefc ^ 

) tr (M„R dGdQ, (68) 



L 2 - L 



fc=l J=l m=l 



which, taking into account X^fe=i R-fcfe = an d defining M = l/L^Li may be rewritten as 



L L 



k—1 m—1 



lit /?(G) E E tr " ( M ^™") dGdQ 

] —— [ [ /3(G) E E tr f M fefc R mm ) tr f(IM - M fefe )R rom ) dGdQ. (69) 
-VJqJg k=lm =l V ' V 7 



L(L 



Now, rearranging terms and summing in k, we get 



L 



L—l 



Q JG 



/3(G) E 



m — 1 



]Ttr 2 (M fefc R 

,fe=i 



tr 2 (MR mm ) 



dGdQ. 



(70) 
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To conclude the proof of this lemma we must solve the above integral. To do so, let us consider the eigenvalue 
decomposition (EVD) of R mm as R mm = V m A m V^ and introduce the change of variables G — > GV m PD 
with P € P a permutation matrix, and D € M ArxAr a diagonal matrix with ±1 entries. Thus, we can write 



r cx [ [ /3(G) ^PD [ j m k ml - mm T ] DP T A r 



dGdQ, 



(71) 



where m = diag(M), = diag(M / r. fc ), and A m = diag(A m ) contains the eigenvalues of R mm . Finally, T may 
be rewritten as 

L 

T cx A ™ EA ™; (72) 



m— 1 



where 



E = PD 



L 



-iLLwixt.**)*'** 



k=l 



DP J 



(73) 



with ej; = rrifc — m, and we have taken into account that J2k=i m k = Lm. Thus, since the value of T does not 
depend on the particular choices of P or D, we have 



yielding 



E = PDEDP J 



r cx ||R 

m— 1 



cl 



N ; 



(74) 



(75) 



The following lemma deals with the second integral in the RHS of p9) . 

Lemma 7: The second integral in the expression of the density ratio can be rewritten as 



L L 



k = l l=k+l 



4^^ M /3(G)Re 2 {tr (M ifc R fc/ ) } dGdQ = c £ ||r m 



k,l=l 
Ij^k 



(76) 



(77) 



(78) 



where c is the same constant as in Lemma [6] 
Proof: Consider the integrals 

®k,l = 4 f Q J G P( G )R g2 (M ;fe R fei ) } dGdQ, 

and the change of variable Q — » QP, with P a permutation matrix. Then, we can write 

©fc.z 1 J j /3(G)Re 2 {tr (m^^jRw) } dGdQ, 

and since the value of the integral cannot depend on the particular choice of the permutation, we may arbitrarily 
select 

®k,l = 4 J q J g P( G )^ 2 {^ (M 12 R fei ) } dGdQ. (79) 
Now, let us rewrite the previous integral as 

& kU = f f /3(G) tr 2 (MR fe! ) dGdQ. (80) 

Jo Jg 
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where 



M = 



Mn M12 
M 22 



R 



kl 



Rki Oa? 



(81) 



We shall continue by applying the change of variable Q <— QU, where U combines the two first columns of Q 
as follows 

U 2X L- 2 

0l-2x2 Ii-2 



u 



(82) 



with 



and induces the transformation 



U 



V2 



1 -1 
1 1 



R kl <- (U ® l N ) H R kl (V ® Ijv) = J 



R fe; + R fei R fei - Rfej 



(83) 



(84) 



The proof now follows the lines in Lemmas [5] and [6] First of all, as a direct consequence of Lemma [5] we may 
decompose ® k .i as 



®k,i = 



■ / / /3(G) tr 2 ([Mn -M M ] 
Jo Jg k 



R k i + R-w 



dGdQ + ^ I I /3(G) tr 2 (MR fc ,)dGdQ, (85) 



with 



R 



(86) 



Moreover, the second integral in the RHS of the previous equation can be simplified by introducing the change of 
variable Q <— QU', with U' combining the two first columns by means of 



U' = 

This change of variable induces the transformation 

R kl <- {V' ®I N ) H R kl (\J' ®I N ) 

which yields 
1 



1 

71 



j -j 
i i 



j(R kl - Rjg) 0,v 

Oat -j Am - ftjg) 



(87) 



(88) 







k.l 



/3(G) tr 2 ([Mn -M 22 ] lAki+Rki 



dGdQ 



/3(G) tr 2 ([Mn - M aa ] j [&«-&«' 



dGrfQ. (89) 



The rest of the proof follows the lines in Lemma [6] The above integrals are almosj^] identical in form to those 
considered in d70) and therefore the same results apply. In particular, we have 



(R ki + Rf{) + j(R k i - Rkl 



2" 








= 2c 


Rkl 



(90) 



5 In this case, the sum of the diagonal blocks is zero, instead of Lljy. However, this does not change the results. 
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where c > is the constant introduced in Lemma [6] Finally, we can sum over all the off-diagonal blocks (instead 
of over those above the main diagonal) to obtain the desired result. ■ 
Finally, the proof of the theorem follows from the direct combination of Lemmas [5] [6] and [7] 
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